294 research outputs found
Can You Follow Me? Testing Situational Understanding in ChatGPT
Understanding sentence meanings and updating information states appropriately
across time -- what we call "situational understanding" (SU) -- is a critical
ability for human-like AI agents. SU is essential in particular for chat
models, such as ChatGPT, to enable consistent, coherent, and effective dialogue
between humans and AI. Previous works have identified certain SU limitations in
non-chatbot Large Language models (LLMs), but the extent and causes of these
limitations are not well understood, and capabilities of current chat-based
models in this domain have not been explored. In this work we tackle these
questions, proposing a novel synthetic environment for SU testing which allows
us to do controlled and systematic testing of SU in chat-oriented models,
through assessment of models' ability to track and enumerate environment
states. Our environment also allows for close analysis of dynamics of model
performance, to better understand underlying causes for performance patterns.
We apply our test to ChatGPT, the state-of-the-art chatbot, and find that
despite the fundamental simplicity of the task, the model's performance
reflects an inability to retain correct environment states across time. Our
follow-up analyses suggest that performance degradation is largely because
ChatGPT has non-persistent in-context memory (although it can access the full
dialogue history) and it is susceptible to hallucinated updates -- including
updates that artificially inflate accuracies. Our findings suggest overall that
ChatGPT is not currently equipped for robust tracking of situation states, and
that trust in the impressive dialogue performance of ChatGPT comes with risks.
We release the codebase for reproducing our test environment, as well as all
prompts and API responses from ChatGPT, at
https://github.com/yangalan123/SituationalTesting.Comment: EMNLP 2023 Main Paper (Camera Ready
High Efficiency Secondary Somatic Embryogenesis in Hovenia dulcis
Embryogenic callus was obtained from mature seed explants on medium supplemented with 2,4-dichlorophenoxyacetic acid. Primary somatic embryos (SEs) can only develop into abnormal plants. Well-developed SEs could be obtained through secondary somatic embryogenesis both in solid and liquid cultures. Temperature strongly affected induction frequency of secondary embryogenesis. Relatively high temperature (30∘C) and germinated SEs explants were effective for induction of secondary somatic embryos, and low temperature (20∘C) was more suitable for further embryo development, plantlet conversion, and transplant survival. Somatic embryos formed on agar medium had larger cotyledons than those of embryos formed in liquid medium. Supplementing 0.1 mg L−1 6-benzyladenine (BA) was effective for plant conversion; the rate of plant conversion was 43.3% in somatic embryos from solid culture and 36.5% in embryos from liquid culture. In vitro plants were successfully acclimatized in the greenhouse. The protocol established in this study will be helpful for large-scale vegetative propagation of this medicinal tree
On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learning
Incorporating contrastive learning objectives in sentence representation
learning (SRL) has yielded significant improvements on many sentence-level NLP
tasks. However, it is not well understood why contrastive learning works for
learning sentence-level semantics. In this paper, we aim to help guide future
designs of sentence representation learning methods by taking a closer look at
contrastive SRL through the lens of isotropy, contextualization and learning
dynamics. We interpret its successes through the geometry of the representation
shifts and show that contrastive learning brings isotropy, and drives high
intra-sentence similarity: when in the same sentence, tokens converge to
similar positions in the semantic space. We also find that what we formalize as
"spurious contextualization" is mitigated for semantically meaningful tokens,
while augmented for functional ones. We find that the embedding space is
directed towards the origin during training, with more areas now better
defined. We ablate these findings by observing the learning dynamics with
different training temperatures, batch sizes and pooling methods.Comment: Accepted by ACL 2023 (Findings, long paper
The prediction of the quality of results in Logic Synthesis using Transformer and Graph Neural Networks
In the logic synthesis stage, structure transformations in the synthesis tool
need to be combined into optimization sequences and act on the circuit to meet
the specified circuit area and delay. However, logic synthesis optimization
sequences are time-consuming to run, and predicting the quality of the results
(QoR) against the synthesis optimization sequence for a circuit can help
engineers find a better optimization sequence faster. In this work, we propose
a deep learning method to predict the QoR of unseen circuit-optimization
sequences pairs. Specifically, the structure transformations are translated
into vectors by embedding methods and advanced natural language processing
(NLP) technology (Transformer) is used to extract the features of the
optimization sequences. In addition, to enable the prediction process of the
model to be generalized from circuit to circuit, the graph representation of
the circuit is represented as an adjacency matrix and a feature matrix. Graph
neural networks(GNN) are used to extract the structural features of the
circuits. For this problem, the Transformer and three typical GNNs are used.
Furthermore, the Transformer and GNNs are adopted as a joint learning policy
for the QoR prediction of the unseen circuit-optimization sequences. The
methods resulting from the combination of Transformer and GNNs are benchmarked.
The experimental results show that the joint learning of Transformer and
GraphSage gives the best results. The Mean Absolute Error (MAE) of the
predicted result is 0.412
- …